How to Make Life More Colorful: From Image Quality to Atmosphere Experience

Image quality and color appearance have been extensively studied in the past decades, which has resulted in high quality displays. Although research on image quality is still ongoing, most improvements have only marginal effects. A new trend in display technology is emerging that focuses on enhancing the overall visual experience of the user. Two features that have been proven to be effective are the introduction of stereoscopic depth and dynamic surround light. In order to further enhance the user’s experience, the atmosphere of the entire room could be adapted to the emotional content of the video. This paper gives a brief overview of research from image quality to the emotional impact of light emitting devices and identifies the research challenges for creating colorful and appealing experiences. Introduction Color is an important aspect of our everyday lives. From an evolutionary point of view, animals with color vision were better suited to gather food, to spot enemies and to pass on their genes. Nowadays, color is used by humans in many areas, like art, architecture, fashion, communication and entertainment. The reason of using color can be very divers, e.g. to draw people’s attention, to transfer information or to create an experience. Since the introduction of an electrical supply network in the late 1800s, products have been developed that emit colored light, such as TVs and lamps. Nowadays, these products are a matter of course. Most households in developed countries have more than one TV, computer, mobile phone or digital camera with a color display. In outdoor spaces, color is used frequently since the introduction of neon lights for signage and city beautification. Whereas in the past light emitting devices were mainly developed for their functional benefits, the emotional value of these devices is becoming more and more important. The image quality of displays has improved drastically over the years, from blurred black and white images to colorful high resolution images. Lighting technology has been improved as well, from inefficient incandescent light bulbs to energy saving compact fluorescent lamps and LEDs. As the functional quality of these devices is reaching the level required by the average user, the next challenge is to optimize the experience of the end-user. This paper gives a brief overview of research from image quality of displays to the emotional impact of light emitting devices. Image Quality Marketing studies consistently show that image quality is one of the most important considerations for consumers to purchase a display besides costs. In order to make high quality displays, display manufacturers need to know how the technology variables of the imaging system, such as the thickness of the color filters or the size of the pixels, affect the image quality as perceived by the end-users. However, assessing the relation between image quality and technology variables appears to be a time consuming and inefficient task, especially because the optimal value of one technology variable usually depends on the values of several other technology variables. This means that the optimization of one display system does not provide knowledge on the optimization of another display system. To overcome this problem, Engeldrum [1] has developed the Image Quality Circle that breaks the relation into three measurable steps (Figure 1). In the first step, image quality is consider to be a multidimensional concept that can be described by a weighted sum of perceptual image quality attributes, such as brightness and sharpness. These attributes can only be determined by human observers and are expressed as perceived strengths (e.g. very bright or very dim). In the second step, each perceptual attribute is related to the physical characteristics of the light emitted by the display, such as the chromaticity of the red, green and blue primaries. In the third step, the physical light output of the display is described as the combination of all technology variables. For instance, changing the thickness of a color filter will affect both the luminance and the chromaticity of the corresponding primary. Image processing algorithms can also be considered as part of the technology variables. The problem of optimizing image quality is now redefined as three questions: (1) what are the image quality attributes and their relative importance for overall image quality, (2) what is the influence of physical characteristics of the light output on the perceptual attributes, and (3) what is the relation between technology variables and the physical light output? Figure 1. The Image Quality Circle model of Engeldrum [1]. Image quality attributes Experimental studies have revealed several perceptual attributes that contribute to the overall image quality of a display system, such as brightness, contrast, color appearance, sharpness and flicker [2]. The relation between these attributes and the overall image quality is, however, far from trivial. One of the 1 2 3 4 Image Quality Circle Image Quality Technology Variables Physical Image Characteristics Image Quality Attributes 17th Color Imaging Conference Final Program and Proceedings 123 reasons is that the relation depends on several factors, such as image content, ambient illumination and personal preference. Moreover, the image quality attributes are usually measured on relative scales (e.g. brighter or sharper) and not on absolute scales that are comparable. Research is underway to express the relative importance of the attributes in terms of the just noticeable difference (JND) of each attribute. In the mean time, qualitative studies have demonstrated that color appearance is one of the most important attributes that naïve viewers use to rank the quality of different high-end TV sets shown next to each other [3]. Color appearance The color appearance of an image presented on a display depends on physical characteristics of the display, but also on characteristics of the surround illumination. In this section only the display will be considered. The range of colors that can be rendered on a display is usually represented by a 3-dimensional shape in a given color space. This so-called ‘color gamut’ is determined by the chromaticity of the display’s primaries, the intrinsic white-point and the gray scale transfer function of each primary. In order to achieve the same color rendering on different displays, video material is encoded according to a standard format (e.g. EBU Tech. 3213 or ITU Rec.709), specifying the primaries, white-point and transfer function, but also the frame rate and resolution. Only displays that comply with the standardized color gamut are able to reproduce colors accurately without additional image processing. Due to technology constraints, displays can have a significantly smaller color gamut compared to the standardized gamut, as is the case for most hand-held devices. In order to provide guidelines for display manufactures, research has determined the variations that are allowed in the chromaticity coordinates of the primaries for the image to be perceived as natural [4]. People are most tolerant for a saturation reduction of the blue primary and much less tolerant for a saturation reduction of the red and green primaries. On the other hand, people are least tolerant for a hue change of the blue primary and more tolerant for a hue change of the green primary. For the red primary, hue changes towards blue are more acceptable compared to hue changes towards green. Also, the white-point of a display does not always correspond to the standardized value of D65. Research has found that deviations of the white-point are more acceptable for variations along the black body curve compared to variations perpendicular to the black body curve [5]. Recent advances in backlight technology of LCDs have made it possible to expand the color gamut towards more saturated primaries. In addition, displays with more than three primaries in a spatial or time-sequential pattern have been proposed. The added value of these wide-gamut displays is based on two observations. First, it is known that not all natural colors can be reproduced within the standardized gamut [6]. Second, people usually prefer colors to be slightly more saturated than what is natural [7]. It has been shown that using the RGB values of an image to directly drive the wide-gamut display can lead to unacceptable colors [8]. For instance, objects at high saturation and high luminance might appear to be fluorescent. Recent studies have determined the maximum gamut size that results in the most preferred or acceptable color rendering, using a large set of complex images [9] or using images containing mainly one hue [10]. Both studies show that the preferred chroma for most images is located outside the EBU gamut, which illustrates the need for wide-gamut displays. The preferred chroma and maximally acceptable chroma were found to depend on image content and personal preference, and, to a lesser extent, on hue. Color processing Once the color gamut of a display is determined, image processing algorithms can be used to change the physical light output for a given RGB input value and, hence, to improve the color appearance. When the (output) gamut of the display is smaller than the (input) gamut of the image, a combination of clipping and scaling is usually applied. Clipping out-of-gamut colors to the borders of the output gamut has the advantage of retaining the saturation of most colors at the expense of losing color detail in areas with high saturation. Scaling of the input gamut has a limited effect on color detail but reduces the saturation of all colors. Both clipping and scaling can be applied in many different ways, e.g. one could change the lightness of the input color, the chroma, the hue or a combination of these color attributes. In the past, many different gamut compression algorithms have been proposed [11]. Image processing is also needed for wide gamut displays in order to avoid over-saturated colors, as mentioned before. A straightforward mapping algorithm can be used to exactly reproduce the colors of the (smaller) standardized input gamut. However, the challenge is to make optimal use of the additional freedom and to deliberately modify colors in order to create images that are more appealing to the user. Again, there are numerous ways to enhance a color. Changing the hue of original colors is usually not appreciated. Therefore, colors should be expanded in the direction of lightness, chroma or a combination of these color attributes. In addition, the extension of colors along a direction can be linear or non-linear. Studies have shown promising results for an adaptive gamut extension algorithm that uses non-linear mapping in a direction depending on the color’s lightness level [8]. Interestingly, a similar algorithm can also be used to improve the preferred color appearance (in contrast to natural color appearance) on displays with a standard EBU gamut. Visual Experience While research on image quality improvements of displays is still ongoing, a new trend emerging for TV displays aims at enhancing the visual experience of the user. One of the reasons for people to watch TV is to relax and to escape reality for a moment. People want to forget their physical space and to have the impression to be part of the displayed space. Research has studied possibilities to enhance people’s visual experience beyond improving image quality. Two features that have been proven to be effective are the introduction of stereoscopic depth and dynamic surround light [12]. Here, only the effect of surround light will be discussed. 124 ©2009 Society for Imaging Science and Technology Effect of surround light First of all, it is well known that the illumination surrounding a display affects the perception of image quality attributes, such as color and contrast. When the illumination level of the surround increases, objects will be perceived as more colorful and the apparent contrast of the image will increase [13]. The chromaticity of the illumination influences color appearance as well, due to chromatic adaptation of the human eye. The adapted white-point, i.e. the chromaticity that is perceived as achromatic, and the preferred white-point of a display both shift towards the chromaticity of the ambient illumination [14]. This means that the blue sky of a displayed image will be perceived as very blue under yellowish light and less blue under bluish light. Color appearance models have been developed to predict the perceived color taking into account effects of the surround. The most recent model recommended by the CIE is CIECAM02 [15]. Secondly, the ambient illumination level has an effect on the visual comfort of the user. Watching TV in a dark room can create physical discomfort caused by the large contrast in luminance between the display and the dark background. The Society of Motion Picture and Television Engineers (SMPTE) recommends an ambient light level of about 10 percent of the peak white output of the TV in order to minimize eye strain for the viewer. Recently, Philips developed a TV, called Ambilight TV, that projects surround light from the back of the TV onto the rear wall (see Figure 2). The level and chromaticity of the surround light can either be static or change dynamically. It has been demonstrated that an Ambilight TV with static white surround light improves the visual comfort of the viewer while watching a 60 min movie compared to a TV without this feature [16]. Surround light had a positive, but modest effect on subjective ratings of visual discomfort, fatigue and eye strain and on physiological measures, such as eye blink frequency, reaction time to visual stimuli and event related potentials. The added value of Ambilight TV is, however, largest when the chromaticity of the surround light changes in accordance with the colors of the video content shown on the display. Dynamic surround light not only reduces visual discomfort but also enhances the visual experience of the user. The benefit of dynamic surround light is related to the fact that the ability to discriminate details decreases with the angular distance from the line of sight [17]. As a consequence, the colored surround gives the impression that the size of the display is extended. It has been found that large displays cause higher physiological arousal and higher subjective ratings of excitement when playing a game compared to small displays [18]. Visual Experience Model In order to study the experience of new display features in a structured way, a framework like the Image Quality Circle is needed. Several studies have shown, however, that image quality is not the appropriate concept to measure the visual experience of display systems. For instance, it has been shown that the perceived image quality of a video sequence shown on an Ambilight TV is equal when the Ambilight is turned on or off [12]. The same has been found for stereoscopic depth. On the other hand, when participants are asked to evaluate viewing experience (i.e. ‘the perceived degree of overall viewing experience’) or presence (i.e. ‘the perceived degree of becoming part of the displayed space’), the effect of Ambilight is highly significant [12]. Both viewing experience and presence were higher for a display with Ambilight compared to a display without Ambilight. Moreover, viewing experience and presence also depended on the perceived image quality of the video sequence, which was varied by changing the compression level. This means that both evaluation criteria are useful to measure the combined effects of image quality and immersive display features such as Ambilight. However, the criteria do not measure the same experience. Image quality has a larger effect on viewing experience than presence, whereas the effect of Ambilight is largest for presence. Based on these results, a new conceptual model can be designed, as depicted in Figure 3. For each factor (e.g. image quality or Ambilight) that influences the evaluation criterion, a Quality Circle can be developed that describes the relation between the factor and technology variables via perceptual attributes and physical characteristics. Figure 2. The Ambilight TV of Philips with dynamic surround light. Quality of Ambilight Knowing that people’s experience can be enhanced by Ambilight, the next question is how to optimize the perceived quality of Ambilight. The first challenge is to find the perceptual attributes that influence the quality of Ambilight. One could imagine that brightness, colorfulness, spatial uniformity and smoothness are relevant attributes, although this has not been scientifically proven yet. The first Ambilight TVs contained CCFL lamps with relatively low saturation but high brightness. Later, RGB LEDs were used to increase the saturation of the surround light. However, LED based lighting systems usually have difficulties in creating a spatially uniform light effect. The reason is that the chromaticity of LEDs varies from sample to sample and the luminance output of LEDs varies over time. Perception studies have been performed to determine the difference in luminance and chromaticity between LEDs that is allowed such that the light effect is perceived as uniform [19]. Another attribute that is assumed to affect the quality of Ambilight is smoothness. Smoothness is defined as the degree in which dynamic light is perceived as continuous. LED based lighting systems use discrete signals to control the light sources, with a limited number of intensity levels per color channel and a limited driving frequency. As a result, the smallest difference between two successive colors is limited, both in color and time. This might lead to perceived discontinuities. Existing spatial color models, such as CIELab, can be used to predict perceived 17th Color Imaging Conference Final Program and Proceedings 125 Figure 3. The model proposed for the evaluation of the visual experience of displays. The evaluation criterion could be e.g. ‘’viewing experience” or “presence”. smoothness of spatial patterns. However, no extensive model on temporal color perception exists. Recent studies have measured the visibility threshold of smoothness, defined as the maximum color difference between successive colors for which the light pattern is perceived as smooth [20]. If thresholds are expressed in CIELab color space (ΔE ab ), people appear to be much more sensitive to discontinuities (which corresponds to a low threshold) when the lightness is varied over time than when the chroma or hue is varied over time. In addition, sensitivity decreases with increasing driving frequency. The sensitivity to temporal smoothness can be described by a simple model. In the model, the smoothness threshold is not expressed as the maximum color difference (ΔE ab ), but as the maximum color difference that is varied per second (ΔE ab * f), also called ‘speed’. The natural logarithm of the maximum speed is a linear function of the driving frequency. The slope of the function is similar for temporal variations in lightness, chroma and hue, whereas the intercept depends on the light attribute. These results can be considered as the first step towards a model on temporal color perception. The model can be used by manufactures to select the right combination of driving frequency and color difference in order to create smooth light effects at a given maximum speed. It can also be used to develop algorithms for Ambilight TV in order to create visually appealing surround light effects that enhance the overall experience of the viewer. Atmosphere experience The high success of Ambilight TV shows again that people enjoy looking at a colorful world. The use of chromatic and dynamic surround light has made watching TV a more colorful experience. The question is now: would it be possible to even further enhance the experience of the viewer or do we have reached the limit? New research projects are investigating if people would feel more immersed when the atmosphere of the entire room matches the emotional content of the video, for instance, a frightening atmosphere when watching a thriller and a cozy atmosphere when watching a happy-family movie. At the same time, new devices are entering the consumer market that enable the design of a large variety of light effects. These devices are based on LED technology, which show many benefits above conventional lighting technologies, such as high saturated primaries, improved spatial and temporal resolution and a small form factor. One example of a device that people can use at home to illuminate their walls with colored light is the Living Colors lamp of Philips. While the technology to create complex light effects is available, it is not known how to make use of the large degree of freedom to create, for instance, a frightening or cozy atmosphere. Extensive research exists on the effects of white light on visibility, task performance and visual comfort [21]. However, relatively few studies have investigated the psychological effects of light. Moreover, research on the effect of light on people’s mood has revealed contradictory results. These discrepancies might be caused by differences in exposure time and/or differences in the methods used to measure mood. On the other hand, since mood is also affected by many other environmental factors (e.g. temperature) and non-environmental factors (e.g. cognition), it is very unlikely that light will always affect a person’s mood to the same extent. Therefore, the concept of atmosphere experience has been introduced. Atmosphere differs from mood in the sense that it is not an affective state but the affective evaluation of the environment. It is the subjective impression of the environment related to the expected effect on mood, but it does not necessarily correspond to the actual mood. Although people might have different opinions about the atmosphere of an environment, it is expected that the effect of light on atmosphere will be more stable than the effect on mood since it is based on people’s experience in the past. A few methodologies have been developed to evaluate people’s impression of an environment. While researchers have used different definitions and terminologies, all studies show that atmosphere is a multidimensional concept. Flynn was one of the first researchers to study the ‘subjective impression’ of an illuminated room [22]. He used a list of semantic differential scales that could be grouped into five factors: perceptual clarity (clearhazy), spaciousness (large-small), evaluative (pleasantunpleasant), privacy (public-private) and relaxation (relaxedtense). However, the first two factors are related to the perception of the illumination and the space, but not to the affective impression. Other researchers have used a two-dimensional bipolar space to describe the ‘affective quality attributed to an environment’ [23]. The two orthogonal dimensions were based on a model to describe mood and emotions and were described as pleasant-unpleasant and arousing-sleepy. However, it has not been tested if the two dimensions are suitable to describe all possible atmospheres. Recently, another method to measure the ‘perceived Image Quality Circle Ambilight Quality Circle ... Evaluation Criterion 126 ©2009 Society for Imaging Science and Technology atmosphere’ has been developed based on a list of 38 unipolar atmosphere scales [24]. The scales were selected from a large list of terms that people use when talking about atmosphere. A statistical analysis revealed that atmosphere can be described by four dimensions: coziness (including items like ‘cozy’, ‘pleasant’ and ‘intimate’), liveliness (including ‘lively’, ‘exciting’ and ‘inspiring’), tenseness (including ‘tense’ and ‘terrifying’) and detachment (including ‘formal’ and ‘business-like’). Atmosphere Experience Circle In order to understand the relation between atmosphere and light, the Quality Circle framework could be used again. Whereas in the models mentioned so far the evaluation criterion varies on a scale from good to bad, the quality of an atmosphere strongly depends on the effect that is desired at a certain moment. Therefore, the aim of the atmosphere model is not to predict quality but to predict the kind of atmosphere that is experienced. As mentioned before, atmosphere is a multidimensional concept. The Atmosphere Experience Circle assumes that each of the atmosphere dimensions are determined by various perceptual properties of the illumination, for instance, brightness impression, color of the illumination, and uniformity of the light distribution. The way in which an observer unconsciously derives the atmosphere from the perceptual light attributes is expected to depend on individual characteristics such as age, gender or culture. The perception of each light attribute is related to the physical light distribution in the room. This relation is thought to be less individual dependent, as it is mainly determined by the properties of the human visual system. Finally, the light distribution depends on technology characteristics of the lighting system, such as lumen output and the optical design of the fixture. This relation is known to a large extent. Complex computer programs can be used to estimate the light distribution based on properties of the light source and reflectance properties of objects in the room. The main research challenges of the Atmosphere Experience Circle are: (1) to find the relation between light attributes and the atmosphere dimensions and (2) to find a number of relevant physical variables that can be extracted from the light distribution and correlates with the perception of light attributes. Effect of general and decorative lighting In our laboratory, several studies have been performed to investigate the first relation of the model [25]. User studies have shown that people do not like to be illuminated by chromatic light, but they like the use of chromatic light to illuminate walls or other objects. Therefore, all light settings that were studied consisted of white light for general lighting, while white or chromatic light was added as decorative lighting in part of the settings. The light attributes that were studied are: brightness impression, color temperature (of general lighting), hue and colorfulness (of decorative lighting) and spatial uniformity. So far, only native Dutch people evaluated the light settings by using the Dutch atmosphere questionnaire [24]. Most of the studies were performed in an empty room of about 6 x 4 x 3 meters with white walls, a gray carpet and various light sources. In one experiment, the room was either furnished as a living room or an office. The effect of the illumination on the atmosphere was found to be independent of the context of the room. This means, for instance, that although a living room might appear to be more cozy than an office with the same light setting, in both situations the atmosphere becomes more cozy when decreasing the color temperature. All studies showed significant and consistent effects of the light attributes on perceived atmosphere. Here, some main effects will be mentioned. For general lighting, coziness was found to be negatively related to brightness, color temperature and perceived uniformity. Liveliness was found to be positively related to brightness and negatively related to color temperature and perceived uniformity. Tenseness was negatively related to brightness and positively related to color temperature and perceived uniformity. Finally, detachment was positively related to brightness, color temperature and perceived uniformity. For decorative lighting, hue had a strong impact on the perceived atmosphere. For instance, yellow and red were perceived as very cozy, whereas cyan was perceived as formal. Interestingly, the effect of red versus blue decorative lighting on perceived atmosphere was similar but larger than the effect of warm white versus cool white general lighting. Whereas these studies used one single chromaticity for decorative lighting, extensive interviews with lighting designers have shown that they prefer to use color combinations to create an atmosphere. For instance, they suggest using orange and blue for a cozy atmosphere and cyan and blue for an activating atmosphere in a living room [26]. So far, no scientific knowledge exists on the effects of combinations of chromatic light on atmosphere. Dynamics is another variable that lighting designers like to use in their designs. For instance, they suggest using slowly changing light in a relaxing atmosphere and faster dynamics for an exciting atmosphere. There are numerous ways to create dynamic light with RGB light sources. For instance, one could change one color attribute (lightness, chroma or hue) over time or combinations of these attributes. In addition, the transition from one color to another color could be a straight line in a given color space, or it could go through white. Once the color transition is determined, the speed can be varied linearly or non-linearly and slow or fast. Currently, not much knowledge is available on the perception and preference of dynamic light for atmosphere creation. It will be a challenging research topic for the near future. Applications There are many opportunities to use current and future knowledge on how to create a desired atmosphere with static and dynamic light. As mentioned, it could be used to further enhance the viewing experience of watching TV. It could also be used to get people in the right mood for other activities at home, such as relaxing or having a drink with friends. In retail, atmosphere creation could assist in attracting more people and increasing sales. In hospitals, it might enhance the wellbeing of patients and accelerate their recovery. And in offices, people might feel more motivated when the lighting is adapted to their emotional needs. In all situations, the illumination has to be adjustable, depending on the video content, mood, activity, time of the day or season. This could be done automatically by an intelligent lighting system or people could change the lighting themselves. The first implementation could be used as an extension of the Ambilight algorithm. In this case, the TV should be able to communicate with at least part of the light sources in the room. This is, however, not easy to realize, but it will probably be possible within several years 17th Color Imaging Conference Final Program and Proceedings 127 from now using more robust wireless communication protocols. Automatic atmosphere creation is not the preferred solution for most other applications, as it is known that people like to be in control of the lighting. However, people are not used to create complex light settings, other then turning light sources on or off or dimming their light output. As the capabilities of new light sources increases, including color and dynamics, it will become more difficult for the end-user to create the desired light effect. Therefore, new ways of changing the lighting are needed. New user interfaces have to be developed that allow people to easily change the entire atmosphere of the room, taking into account the right balance between complexity and flexibility. Future research challenges The transition from image quality to atmosphere experience with light emitting devices has introduced many interesting topics that require more research. One of the challenging topics for image quality is color enhancement and the development of gamut extension algorithms. Ambilight TV has introduced the need for models on temporal color perception and (more) knowledge on the effect of surround light on the color appearance on a display. In order to quantify the effect of light on atmosphere creation, knowledge is needed on chromatic adaptation and accurate models have to be developed that predict the color appearance for related colors. Another interesting question is how to create appealing dynamic light effects for different applications. In the Atmosphere Circle Model knowledge is required to predict the perception of light attributes, such as room brightness and uniformity, from the physical light distribution. In addition, a new measurement method is needed to quantify the physical light distribution and to extract meaningful variables that correlate with the perception of the light. All these topics will help to create not only a functional but also an appealing environment.